Systems, methods, and apparatuses for creating a shared file system between a mainframe and distributed systems

ABSTRACT

The disclosed system may comprise a mainframe computing resource, a data library, a data processing appliance, and a distributed system. The data library may be securely connected to the mainframe computing resource. The data library may be configured to receive data from the mainframe computing resource via a first interface. The data processing appliance may be configured to read and write data to the data library via a second user interface. The distributed system may be configured to receive data from the data processing appliance. The distributed system may be further configured to process the data based on a workflow from the mainframe computing system.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a reissue application of U.S. Pat. No. 9,898,484issued on Feb. 20, 2018 (aka U.S. Ser. No. 14/822,770 filed on Aug. 10,2015) entitled “Systems, Methods, and Apparatuses for Creating a SharedFile System Between A Mainframe And Distributed Systems.” The '484patent is a continuation of, claims priority to and the benefit of, U.S.Pat. No. 9,898,483 issued Feb. 20, 2018 (aka U.S. Ser. No. 14/822,740filed Aug. 10, 2015) entitled “Systems, Methods, and Apparatuses forCreating a Shared File System Between a Mainframe and DistributedSystems,.”the entire contents of which is hereby incorporated byreference The entire contents of both patents are hereby incorporatedherein by reference for all purposes.

FIELD

The present disclosure relates to distributed data processing systems,and more particularly, to creating a shared file system between amainframe computing resource and distributed systems.

BACKGROUND

Typical enterprise data processing systems that include mainframes anddistributed applications leverage FTP and direct connect protocols tomove data between the mainframe and distributed applications. Thismethod of data transfer and/or data sharing may be inefficient and slow.Moreover, this method of data transfer and/or data processing mayrequire significant mainframe resources which are expensive. As such,there is a need to improve data processing capabilities and to minimizemainframe resource utilization. These improvements may be accomplishedby creating a file sharing and data transfer system that is capable ofquickly exchanging and processing data between mainframe and distributedenvironments.

SUMMARY

In various embodiments, a method of sharing data between a mainframecomputing resource and a distributed system is provided. The method mayinclude operations and/or steps comprising writing, by a mainframecomputing resource, data to a data library. The method may furthercomprise indicating, by the mainframe computing resource, to a dataprocessing appliance that there is data that is ready for processing.The method may also comprise removing, by the data processing appliance,mainframe tape information from the data. The method may furthercomprise processing, by at least one of the data processing appliance orthe distributed system, the data. The method may also comprise writing,by the data processing appliance, processed data to the data library ina mainframe tape format. The method may further comprise notifying, bythe data processing appliance, the mainframe computing resource that theprocessing data is ready for further processing by the mainframecomputing resource. The notification may include at least one of asuccess return code or failure return code

In various embodiments, the method may further comprise indicating, bythe mainframe computing resource, to the data processing appliance aworkflow to use for the data. The method may also comprise reading, bythe data processing appliance, the data from the data library. Themethod may further comprise scheduling, by the mainframe computingresource, a job to process the processed data based on the return code.

In various embodiments, the method may further comprise converting, bythe data processing appliance, the data from a first format to a secondformat. The first data format may be a mainframe communication protocol.

In various embodiments, the data processing appliance may comprise aplurality of programmable logic gates. The mainframe computing resourcemay be configured to communicate with the data library through a firstinterface. The data processing appliance may be configured tocommunicate with the data library through a second interface. The firstinterface may be isolated form the second interface.

In various embodiments, a system may comprise a mainframe computingresource, a data library, a data processing appliance, and a distributedsystem. The data library may be securely connected to the mainframecomputing resource. The data library may be configured to receive datafrom the mainframe computing resource via a first interface. The dataprocessing appliance may be configured to read and write data to thedata library via a second user interface. The distributed system may beconfigured to receive data from the data processing appliance. Thedistributed system may be further configured to process the data basedon a workflow from the mainframe computing system.

In various embodiments, the data processing appliance may be configuredto convert the data from the data library, from a mainframe environmentformat to a distributed system environment format.

In various embodiments, the data processing appliance may be furtherconfigured to receive processed data from the distributed system inresponse to the processing, and to load the data to the data library.The processed data may be stored in a first portion of the data librarydesignated for the data processing appliance. The first portion may besegregated from a second portion of the data library that receives datafrom the mainframe computing resource. The processed data from the dataprocessing appliance may include at least one of a success return codeor failure return code.

The forgoing features and elements may be combined in variouscombinations without exclusivity, unless expressly indicated hereinotherwise. These features and elements as well as the operation of thedisclosed embodiments will become more apparent in light of thefollowing description and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The subject matter of the present disclosure is particularly pointed outand distinctly claimed in the concluding portion of the specification. Amore complete understanding of the present disclosure, however, may beobtained by referring to the detailed description and claims whenconsidered in connection with the drawing figures, wherein like numeralsdenote like elements.

FIG. 1 is a schematic diagram that illustrates a mainframe and midrangecomputing system, in accordance with various embodiments;

FIG. 2 is a schematic diagram that a mainframe, midrange, anddistributed computing system, in accordance with various embodiments;

FIG. 3 is a schematic diagram that a redundant mainframe, midrange, anddistributed computing system, in accordance with various embodiments;

FIG. 4 is a process flow for operating a shared filed system between amainframe, midrange, and distributed computing system, in accordancewith various embodiments; and

FIG. 5 is a process flow for operating a shared filed system with a dataprocessing appliance, in accordance with various embodiments.

DETAILED DESCRIPTION

The detailed description of exemplary embodiments herein makes referenceto the accompanying drawings and pictures, which show variousembodiments by way of illustration. While these various embodiments aredescribed in sufficient detail to enable those skilled in the art topractice the disclosure, it should be understood that other embodimentsmay be realized and that logical and mechanical changes may be madewithout departing from the spirit and scope of the disclosure. Thus, thedetailed description herein is presented for purposes of illustrationonly and not of limitation. For example, the steps recited in any of themethod or process descriptions may be executed in any order and are notlimited to the order presented. Moreover, any of the functions or stepsmay be outsourced to or performed by one or more third parties.Furthermore, any reference to singular includes plural embodiments, andany reference to more than one component may include a singularembodiment.

Mainframe computing resources are typically optimized for bulk dataprocessing and transactional processing. Mainframe computing resourcesare also used for high throughput processing. Traditional mainframes(e.g., IBM zSeries mainframes) can share data via a transactionalservice that shares one or more records with the distributed environmentby exchanging messages over a message queue, web service, and/or otheron line interface. Files may also be transmitted between systems usingfile transfer protocols such as FTP or Direct Connect. Alternatively,the distributed system can be mounted on the mainframe storage system toaccess files stored on the mainframe system. However, in thisarrangement, the distributed system may require full administrativeaccess to the mainframe storage system.

Providing full administrative access to servers in the distributedenvironment may be viewed as an unacceptable security risk to themainframe. In this regard, the security systems that protect mainframedata are separate and user credentials cannot be shared easily.Moreover, providing the distributed environment with direct access tothe mainframe file system to share files (using protocols such asNetwork File System (NFS)) is not typically considered secure.

Mainframe systems may also use communication protocols or data formatsthat are different than the communication protocols or data formats usedby the distributed system. For example, the IBM zSeries typically usesExtended Binary Coded Decimal Interchange Code (EBCDIC) bit characterencoding, while distributed system components and their associatedoperating systems (e.g., Linux operating systems, Windows operatingsystems, and/or the like) may use American Standard Code for InformationInterchange (ASCII). The mainframe operating system supports automatictranslation between EBCDIC and ASCII character sets when transmittingfiles, but there are often issues when translating a packed decimalbetween EBCDIC and ASCII. Both EBCDIC and ASCII represent charactersusing 8 bits to create a byte. Thus, EBCDIC and ASCII represents anumber as an 8 bit byte. A packed decimal encodes two decimal digitswithin a single byte by encoding the numbers 0 to 9 as four bits. Whentranslating between EBCDIC and ASCII, the translator can make mistakesbecause it does not always correctly understand if a byte beingtranslated is a “normal” character or two packed decimal numbers.

In various embodiments, access to mainframe files by distributed systemsmay require translation since the format of the mainframe file isdifferent than the format of the data used and there is no commonstorage management system which both the mainframe and the distributedsystem can access without actually sending the file (FTP) and convertingthe character encoding.

In various embodiments, the file sharing and processing appliance asdescribed herein may be configured to directly and seamlessly integratewith mainframe processing architecture. The data processing appliancemay be configured to transform, share, protect, stream, and/or otherwiseprocess enterprise data stored on the mainframe system. In this regard,the data processing appliance may reduce or eliminate the need totransfer and/or copy the data to be processed by the distributed systemvia an FTP protocol. Moreover, because the data processing appliance isconfigured to receive and process data in real time, the data processingappliance may process mainframe data as it is received from themainframe, rather than waiting for data processing to begin until a copyvia FTP protocol has been accomplished.

In various embodiments, the data processing appliance as describedherein may be configured to receive and transmit data from and/or to anenterprise mainframe system. Moreover, the data processing appliance maybe configured to receive and/or transmit information to productiondatabases, enterprise data warehouses, cloud computing systems, big dataenvironments (e.g., HDFS), application servers, streaming data sources,and/or the like. Moreover, the data processing appliance may beconfigured to offload mainframe processing to obtain an 80-90% reductionin mainframe computing costs. In this regard, the data processingappliance may be capable of more efficiently and effectively processingdata to reduce the consumption of mainframe data resources and/ormainframe computing resources.

In various embodiments, and with reference to FIG. 1, system 100 may becapable of creating a shared file system between a mainframe computingresource 110 and distributed computing environments. By creating ashared file system, file processing can be performed in mainframeenvironments or distributed environments based on the desired economicsand performance. Mainframe computing resource 110 may be configured toprocess files with high throughput rates. However, mainframe computingmay be much more expensive than performing the same processing in thedistributed environment.

In various embodiments, system 100 may comprise a mainframe computingresource 110 and a data processing appliance 120. Mainframe computingresource 110 may be operatively coupled to and in electroniccommunication with data processing appliance 120. For example, mainframecomputing resource 110 may be connected to data processing appliance 120via a secure local area network (LAN). In this regard, mainframecomputing resource 110 may be hardwired via a secure LAN to dataprocessing appliance 120. Moreover, data processing appliance 120 may bein electronic communication with a distributed system 140. Distributedsystem 140 may be any suitable data processing system that is configuredto process mainframe data. For example, distributed system 140 may be areal-time data processing system that is configured to receive real-timedata from mainframe computing resource 110 via data processing appliance120. In this regard, distributed system 140 may be able to stream datafrom mainframe computing resource 110 via data processing appliance 120removing the typical data transfer and copy step via an FTP protocol.

In various embodiments, data processing appliance 120 may be configuredas a backup storage device to share files between mainframe computingresource 110 and distributed system 140. Mainframe computing resource110 may comprise and/or be configured with an interface to dataprocessing appliance 120 and/or distributed system 140. Mainframecomputing resource 110 may also comprise a separate first interface fordata processing appliance 120 and a second interface for distributedsystem 140.

In various embodiments, the backup implementation may employ allintermediary library. The intermediary library may be connected tomainframe computing resource 110 using a high performance networkconnection and distributed system 140 may be connected using an Ethernetinterface such as, for example, TC/IP. Mainframe computing resource 110may be configured to write to the intermediary library and/ordistributed system 140. In this regard, distributed system 140 and/ordata processing appliance 120 may be configured to and/or capable ofremoving the tape semantics created when mainframe computing resource110 writes to intermediary library and is able to sort the file. Dataprocessing appliance 120 may write the sorted file back to theintermediary library so it could be read by mainframe computing resource110. This functionally may reduce the processing cost by using lowercost distributed servers to sort.

In various embodiments, data processing appliance 120 may be configuredto process data received from mainframe computing resource 110. In thisregard, data processing appliance 120 may comprise a plurality ofprogrammable logic gates that are configured to receive and processmainframe data in real time and/or in a batch configuration depending onthe processing requests and programming of data processing appliance120. Given the logic gate architecture of data processing appliance 120,real-time and/or batch processing of mainframe data from mainframecomputing resource 110 may be possible with data processing appliance120 that would otherwise not be possible with traditional legacymainframe computing systems.

In various embodiments, data processing appliance 120 may be configuredto perform in-line processing including, for example, datatransformation (e.g., converting between formats such as EBCDIC andASCII, restructuring data structures such as relational and XML, and/orflattening records such as SMF and EDI). Data processing appliance 120may be further configured to include operations including, for example,sorting (e.g., using complex keys), joining (e.g., merging data),aggregation (e.g., over various data sets and windows), deduplication,data validation (e.g., based on expressions, data types, keywords,and/or the like), key generation (e.g., may incorporate multiplefields), masking (e.g., especially format-preserving), filtering (e.g.,based on keywords, errors, sensitive data, and/or the like), and/or thelike.

In various embodiments, the logic gates of data processing appliance 120may be configured to program the silicon to be programmed directly. Inthis regard, the logic gates are configured to provide hardware speedprocessing operations, with no overhead of the general purpose computermoving around the code to get increased programming efficiency.Moreover, parts of the work flow being carried out data processingappliance 120 can be translated or programed into the logic gates. Forexample, translation of data between a mainframe format and adistributed format may be carried out via the logic gates. Moregenerally, the workflow may be partitioned into jobs that can beperformed by the parallel architectures in data processing appliance 120such as, for example, the logic gates, CPUs, and GPUs of data processingappliance 120.

In various embodiments, data processing appliance 120 may be anysuitable data processing appliance such as, for example, a streamprocessing appliance sold by VelociData®. Data processing appliance 120may be an engineered system that is configured to achieve extremeprocessing performance through the use of massively parallel computerarchitectures that incorporates CPUs, FPGAs and GPUs. Each appliance maybe a 4U rack-mounted system that houses data processing appliance 120components together in an engineered system.

In various embodiments, system 100 may be secured. System 100 may besecured with two interfaces on the intermediary library, such as forexample, a first interface for mainframe computing resource 110 and asecond interface for distributed system 140. By including separateinterface, mainframe computing resource 110 and/or distributed system140 may be configured with different permissions for files. Moreover,the separate interfaces may provide mainframe computing resource 110and/or distributed system 140 with access to different portions and/orareas of the intermediary library.

In various embodiments, in addition to or in lieu of an intermediarylibrary, system 100 may also comprise or be disposed on a shared storageframe used for normal data processing. In this regard, the data to beprocessed does not have to be created on mainframe computing resource110 for processing on mainframe computing resource 110. In this regard,distributed system 140 may also initiate a processing job by writing adata to the intermediary library to be used by mainframe computingresource 110.

In various embodiments, data processing appliance 120 may be located ina midrange environment. Moreover, data processing appliance 120 may beconfigurable with a common file system that is capable of reading andwriting data from and to the mainframe computing resource 110. In thisregard, data processing appliance 120 may be configured to receive datafrom mainframe computing resource 110 in a mainframe computing format.Data processing appliance 120 may be further configured to convert thedata from the mainframe format to a midrange and/or distributed systemformat. In response to the conversion, data processing appliance 120and/or distributed system 140 may analyze, transform, manipulate, and/orotherwise process the converted data. The processed data may then beconverted by mainframe processing appliance from the midrange ordistributed format to a mainframe format that is readable and storableby mainframe computing resource 110. The converted main data in themainframe format may be communicated from data processing appliance 120to mainframe computing resource 110. In this regard, data processingappliance 120, in response to a data processing operation beingcomplete, may create a job for mainframe computing resource 110 thatindicates that the data has been processed by data processing appliance120 and/or distributed system 140 and is ready to be transmitted fromdata processing appliance 120 to mainframe computing resource 110. Thejob created by data processing appliance 120 may open a gate betweenmainframe computing resource 110 and data processing appliance 120 toaffect the transfer of the processed data from data processing appliance120 to mainframe computing resource 110.

In various embodiments, and with reference to FIG. 2A, system 200 maycomprise a mainframe computing resource 210, a data processing appliance220, and a data library 230. Data library 230 may be configured toconnect directly to mainframe computing resource 210 via a high-speedchannel. In this regard, the high-speed channel may be a secure channelthat is configured to transmit and receive mainframe data via datalibrary 230. Moreover, data processing appliance 220 may be operativelycoupled to and in electronic communication with data library 230 via ahigh-speed direct connection.

In various embodiments, data library 230 may be any suitable datastorage and back-up structure. Data library 230 may be an EMC DLm DiskLibrary for mainframe (DLm). Data library 230 may include an enhancedvirtual tape engine and FICON adapters that are configured to operate at8 Gb/s. Data library 230 may comprise and/or be configured withhierarchical storage management (HSM), archiving, and general worktapes. This technology allows for faster backups and it reduces the timefor data recalls from minutes to seconds. Data library 230 may also beintegrated into data processing appliance 220.

In various embodiments, from the mainframe perspective, data processingappliance 220 and/or data library 230 may be detected and/or viewed as atape backup device. In this regard, data processing appliance 220 and/ordata library 230 can be written and read by both mainframe computingresource 210 and distributed system 240. Mainframe computing resource210 has traditionally used tape backup devices, along with read andwrite files. Data processing appliance 220 may be configured as anintegration service or engine between data library 230 and distributedsystem 240. In this regard, data processing appliance 220 operating asthe integration service may normalize the files created by mainframecomputing resource 210, on the back up device, so they appear as normaldistributed files to distributed system 240.

In various embodiments, data processing appliance 220 may be configuredto expose and preprocess data from mainframe computing resource 210and/or data library 230 to the midrange through FTP protocols and/or NSFmounts to create a shared file system with the mainframe allowing forreal-time computing and processing of mainframe data. This configurationmay reduce the computing requests, jobs, and/or computing resourcesconsumed in mainframe computing resource 210 by distributing processingresponsibilities and/or jobs that would otherwise be performed bymainframe computing resource 210 to distributed system 240 via datalibrary 230 and/or data processing appliance 220.

In various embodiments, data processing appliance 220 may be configuredas a persistent file sharing system. In this regard, data processingappliance 220 may be configured to manage the data stored locally ondata processing appliance 220 and/or data library 230 for the variousresources consuming the data including, for example, mainframe computingresource 210 and/or distributed system 240. In this regard, dataprocessing appliance 220 may be configured to selectively provide filesemantics, data fields and encoding to distributed system 240 and/ormainframe computing resource 210 in the format and with the fieldsrequested by each processing appliance (e.g., distributed system 240,mainframe computing resource 210, and/or the like).

In various embodiments, data library 230 may be configured to appear asa stored tape file to mainframe computing resource 210. In this regard,data library 230 may be perceived by mainframe computing resource 210 asa secure tape backup and/or secure FTP repository for data frommainframe computing resource 210. Moreover, given the strict datatransfer protocols and guidelines governing data leaving and/or enteringmainframe computing resource 210, data library 230 may be a stand-aloneand/or integrated system. In this regard, data library 230 may be astand-alone system with a direct connection to mainframe computingresource 210. Moreover, data library 230 may be an integral system indata processing appliance 220 that is directly connected to mainframecomputing resource 210.

In various embodiments, data library 230 may be configured to act as aqueue for data processing appliance 220. In this regard, mainframecomputing resource 210 may be configured to continuous write and/orselectively write to data library 230 in response to receiving jobs totransmit data for processing by data processing appliance 220,distributed system 240, and/or any other suitable job indicating datatransfer for processing. In this regard, data library 230 may beconfigured to act as a queue for mainframe computing resource 210. Datalibrary 230 may buffer jobs to be completed by data processing appliance220 with data provided by mainframe computing resource 210. For example,where data processing appliance 220 is processing data at the timemainframe computing resource 210 receives a job to transmit a secondsubset of data to data processing appliance 220 for processing, datalibrary 230 may receive the second set of data from mainframe computingresource 210 and may store that data until data processing appliance 220has finished processing the first set of data and is capable and/orready to process the second set of data provided by mainframe computingresource 210. In this way, data library 230 may be configured to act asa queue to prevent mainframe computing resource 210 from timing out inresponse to a job being created for data processing appliance 220 whendata processing appliance 220 is already processing a second job.

In various embodiments, and with reference to FIG. 3, mainframecomputing resource 310 may be operatively configured to a redundant dataprocessing system. In this regard, system 300 may comprise a pluralityof data libraries 330 such as, for example, data library 330-1, datalibrary 330-2, and/or the like. Moreover, system 300 may comprise aplurality of data processing appliances 320 such as, for example, dataprocessing appliance 320-1 and data processing appliance 320-2, and/orthe like. The data library 330-1 and/or data library 330-2 may beredundantly connected to mainframe computing resource 310. Similarly,data processing appliance 320-1 and/or data processing appliance 320-2may be redundantly connected to and/or may comprise redundant datalibrary 330-1 and/or redundant data library 330-2 respectively. In thisregard, processing jobs handled by data processing appliance 320-1 maybe seamlessly handled by data processing appliance 320-2 in response todata processing appliance 320-1 experiencing a failure, power loss,and/or other issue limiting the data processing capabilities of dataprocessing appliance 320-1. Similarly, data library 330-1 and/or datalibrary 330-2 may seamlessly handle workload provisioned to the otherdata library in response to data library 330-1 and/or data library 330-2experiencing a power failure, processing failure, and/or any othersuitable failure preventing data library 330-1 and/or data library 330-2experiencing a failure.

In various embodiments and with reference to FIG. 3 and FIG. 4, system300 may be configured to perform a method 400 of processing mainframedata on distributed system 340. Method 400 may comprise writing, bymainframe computing resource 310, data to data library 330 (Step 401).Method 400 may further comprise indicating, by mainframe computingresource 310, to data processing appliance 320 that there is data thatis ready for processing (Step 403). The indicating may be accomplishedby a message sent by mainframe computing resource 310. The indicatingmay also be accomplished by setting a flag by mainframe computingresource 310 that is detectable or monitored by at least one of dataprocessing appliance 320 and/or data library 330. In response tomainframe computing resource 310 creating a file on data library 330,mainframe computing resource 310 may create a place on the storagemedium of mainframe computing resource 310 (e.g., mainframe tapecatalog) for processed data. This configuration and data managementarrangement creates a level a security by separating distributed system340 and the mainframe computing environment.

In various embodiments, method 400 may further comprise indicating, bymainframe computing resource 310, to data processing appliance 320 theworkflow to use for the data (Step 405). Mainframe computing resource310 may send a message to the data processing appliance 320 with thelocation of the data to be processed (e.g., the tape number), thelocation where the data will be stored post processing (e.g., thelocation of the data such as, for example, the tape number of the tapeto be created on data library 330 by data processing appliance 320), andthe workflow to be executed by data processing appliance 320 totransform the data.

In various embodiments, method 400 may further comprise reading, by dataprocessing appliance 320, the data from the data library 330 (Step 407).The indication from mainframe computing resource may indicate to dataprocessing appliance 320 the particular location of the data to beprocessed on data library 330. Method 400 may further comprise removing,by data processing appliance 320, mainframe tape information from thedata (Step 409). In this regard, data processing appliance 320 maypreprocess the data to make the data suitable conversion and/orprocessing by distributed system 340.

In various embodiments, method 400 may further comprise converting, bydata processing appliance 320, the data from a first format to a secondformat (Step 411). In this regard, data processing appliance 320 may beconfigured to translating data from, for example, EBCDIC to ASCII. Dataprocessing appliance 320 may also be configured to unpack packeddecimal, or change packed decimal to an ASCII packed decimal format.Data processing appliance 320 may be further configured to changevariable length fields to fixed length fields, removing tape semantics,and/or the like.

In various embodiments, method 400 may further comprise processing, byat least one of data processing appliance 320 or distributed system 340,the data (Step 413). The data may also be processed by both dataprocessing appliance 320 and distributed system 340. In variousembodiments, data may originate from and/or be created in distributedsystem 340 which may then be transmitted to mainframe computing resource310 through data processing appliance 320 and/or data library 330.

In various embodiments, method 400 may further comprise writing, by dataprocessing appliance 320, processed data to data library 330 in amainframe tape format. In response to the data being processed, dataprocessing appliance 320 may be configured to convert, translate and/orotherwise format the data processed by distributed system 340 into aformat that is suitable for the mainframe environment. To minimize theeffect of corrupt data being mounted to data library 330, dataprocessing appliance 320 may write the processed data to a segregatedstorage area or a dedicated set of disks of data library 330. Dataprocessing appliance 320 may use a dedicated interface, if the interfaceis a file mount by data processing appliance 320. Data processingappliance 320 may also write to a range of tape numbers that are adedicated storage location for data created and/or processed by dataprocessing appliance 320 and/or distributed system 340.

In various embodiments, method 400 may further comprise notifying, bythe data processing appliance, the mainframe computing resource that theprocessing data is ready for further processing by the mainframecomputing resource, wherein the notification includes at least one of asuccess return code or failure return code (Step 417). In this regard,data processing appliance 320 may send a message to mainframe computingresource 310 indicating that the processed data is ready for furtherprocessing and/or storage in the mainframe environment. The indicationor message may comprise the name of the processed data, and a returncode on the success of the workflow executed by the data processingappliance 320.

In various embodiments, method 400 may further comprise scheduling, bymainframe computing resource 310, a job to process the processed databased on the return code (Step 419). In response to the workflow beingsuccessful, the catalog of mainframe computing resource 310 may beupdated with the name or the processed data (e.g., the file). The changeto the catalog may allow mainframe computing resource 310 to continue toread the file from data library 330.

In various embodiments, mainframe computing resource 310 may also beconfigured to communicate with and/or provide data to data processingappliance 320 via an FTP data transfer protocol. In response toreceiving the data via the FTP transfer, data processing appliance 320may execute the prescribed workflow on the data and transfer theprocessed data hack to mainframe computing resource 310. This processwill likely take longer to process data than using system 300 with adata library 330, because the data transfer from mainframe computingresource 310 has to write the file before transmitting.

In various embodiments, data processing appliance 320 may be configuredto allow distributed system 340 to read the data directly using a filesystem protocol such as, for example, network file system (NFS). Thisarraignment may remove the overhead of transmitting the file with aprotocol such as FTP, while still allowing data processing appliance 320to remove the tape semantics and reformat the file for the distributedapplication. The file can write back to data processing appliance 320using NFS to send the file back to mainframe computing resource 310.

In various embodiments and with reference to FIG. 5, method 500 maycomprise receiving, by the data processing appliance, a workflow from atleast one of a mainframe computing resource or a distributed computingresource (Step 5010. The workflow, job, and/or data associated with theworkflow may originate from the mainframe computing resource, thedistributed computing resource, and/or the data processing appliance. Asdiscussed herein, the data processing appliance may comprise and/or beconfigured with a plurality of hardware accelerators and/or a pluralityof computer processing units. The plurality of hardware accelerators mayinclude for example, programmable logic gates, graphic processing units,and/or the like. Moreover, the data processing appliance may beconfigured to partition or parse data received during a job for parallelprocessing over portions of the plurality of hardware acceleratorsand/or the plurality of central processing units. In this regard, thedata processing appliance may be configured to optimize the processingand the speed at which the data is processed by the data processingappliance.

In various embodiments, method 500 may further comprise receiving, bythe data processing appliance, data from at least one of the mainframecomputing resource or the distributed computing resource (step 503).Data from the mainframe computing resource may be formatted in a firstmainframe data format as discussed herein. Data from a distributedcomputing resource may be configured in a distributed computing formatas discussed herein. Method 500 may further comprise preparing, by thedata processing appliance and in real time as the data is received, thedata for at least one of the mainframe computing resource and thedistributed computing resource (step 505). The preparing may includedata being converted to a first format and/or a mainframe data format inresponse to the data being transmitted to the mainframe computingresource. That preparing may also include preparing data or convertingdata to a second format and/or a distributed computing format inresponse to the data being transmitted to the distributed computingresource.

In various embodiments, method 500 may further comprise streaming by thedata processing appliance at contemporaneously with the preparing, thedata for at least one of the mainframe computing resource and thedistributed computing resource (step 507). The streamed data may bestreamed in a first format and/or a mainframe data format in response tothe data being streamed to the mainframe computing appliance. Moreover,the data may be streamed in a second format and/or a distributedcomputing format in response to the data being streamed to thedistributed computing resource.

In various embodiments, method 500 may further comprise receiving by thedata processing appliance, processed data from the distributed computingresource in a second format and/or a distributed computing format (step509). Method 500 may also comprise converting, by the data processingappliance the processed data from the second format to a first format(e.g., a mainframe data format) (step 511). Method 500 may furthercomprise writing, by the data processing appliance and contemporaneouslywith the converting, the processed data to at least one of a datalibrary or the mainframe computing resource (Step 513).

In various embodiments, method 500 may also comprise receiving, by thedata processing appliance processed, data from the mainframe computingresource m a first format (e.g., a mainframe computing format) (step515). Method 500 may further comprise converting, by the data processingappliance the processed data from the first format to a second format(e.g., a distributed computing format) (step 517). Method 500 mayfurther comprise writing, by the data processing appliance andcontemporaneously with the converting, the processed data to at leastone of a data library and the distributed computing resource (step 519).In this regard, the data processing appliance is configured to optimizethe way data is received, conditioned, processed, and then returned toits source. Moreover, the data processing appliance is configured tooptimize the way that data is routed and processed between mainframesand distributed systems.

In various embodiments, the systems and methods described herein mayincrease the efficiency and lower the cost of mainframe computing jobs.In this regard, the systems and methods described herein improve theperformance of the various hardware components of distributed computingsystems with mainframe, midrange and distributed components. Morespecifically, the systems and methods described herein reduce the timespent waiting for file to be physically transmitted between mainframecomputing resource 310 and distributed system 340.

In various embodiments, the systems and methods described herein mayreduce mainframe computing cost. In this regard, writing to data library330 may cost a fraction of the cost to use FTP. Moreover, the systemsand methods described herein are adaptable to various mainframe,midrange, and distributed environments allowing for enterprise widestandardization. Moreover, conventional encryption and decryptionprocesses can be avoided where the connections of the systems describedherein are hardwired.

In various embodiments, the systems and method described herein mayreduce load on an enterprise network. Moreover, applications operatingon distributed system 340 may NFS mount into data processing appliance320 to read mainframe data directly and/or to perform processingoperations including, for example, domain data validation, dataconversion from fixed format to XML, sort-merge-split operations, datamasking and encryption, load data to databases, offload extracttransform load (ETL) processing, prepare data fur big data environments,and/or the like.

As used herein, “match” or “associated with” or similar phrases mayinclude an identical match, a partial match, meeting certain criteria,matching a subset of data, a correlation, satisfying certain criteria, acorrespondence, an association, an algorithmic relationship and/or thelike. Similarly, as used herein, “authenticate” or similar terms mayinclude an exact authentication, a partial authentication,authenticating a subset of data, a correspondence, satisfying certaincriteria, an association, an algorithmic relationship and/or the like.

In various embodiments, the methods described herein are implementedusing the various particular machines described herein. The methodsdescribed herein may be implemented using the below particular machines,and those hereinafter developed, in any suitable combination, as wouldbe appreciated immediately by one skilled in the art. Further, as isunambiguous from this disclosure, the methods described herein mayresult in various transformations of certain articles.

For the sake of brevity, conventional data networking, applicationdevelopment and other functional aspects of the systems (and componentsof the individual operating components of the systems) may not bedescribed in detail herein. Furthermore, the connecting lines shown inthe various figures contained herein are intended to represent exemplaryfunctional relationships and/or physical couplings between the variouselements. It should be noted that many alternative or additionalfunctional relationships or physical connections may be present in apractical system.

The various system components discussed herein may include one or moreof the following: a host server or other computing systems including aprocessor for processing digital data; a memory coupled to the processorfor storing digital data; an input digitizer coupled to the processorfor inputting digital data; an application program stored in the memoryand accessible by the processor for directing processing of digital databy the processor; a display device coupled to the processor and memoryfor displaying information derived from digital data processed by theprocessor; and a plurality of databases. Various databases used hereinmay include: client data; merchant data; financial institution data;and/or like data useful in the operation of the system. As those skilledin the art will appreciate, computer on the distributed system mayinclude an operating system (e.g., WINDOWS®, UNIX®, LINUX®, SOLARIS®,MacOS, and/or the like) as well as various conventional support softwareand drivers typically associated with computers.

The present systems or any part(s) or function(s) thereof may beimplemented using hardware, software or a combination thereof and may beimplemented in one or more computer systems or other processing systems.However, the manipulations performed by embodiments were often referredto in terms, such as matching or selecting, which are commonlyassociated with mental operations performed by a human operator. No suchcapability of a human operator is necessary, or desirable in most cases,in any of the operations described herein. Rather, the operations may bemachine operations. Useful machines for performing the variousembodiments include general purpose digital computers or similardevices.

In fact, in various embodiments, the embodiments are directed toward oneor more computer systems capable of carrying out the functionalitydescribed herein. The computer system includes one or more processors,such as processor. The processor is connected to a communicationinfrastructure (e.g., a communications bus, cross over bar, or network).Various software embodiments are described in terms of this exemplarycomputer system. After reading this description, it will become apparentto a person skilled in the relevant art(s) how to implement variousembodiments using other computer systems and/or architectures. Computersystem can include a display interface that forwards graphics, text, andother data from the communication infrastructure (or from a frame buffernot shown) for display on a display unit.

Computer systems described herein also include various memories such as,for example, random access memory (RAM), a hard disk drive, a removablestorage drive, a magnetic tape drive, an optical disk drive, and/or thelike. The removable storage drive reads from and/or writes to aremovable storage unit in a well-known manner. A removable storage unitrepresents a magnetic tape, optical disk, and/or the like which is readby and written to by a removable storage drive. As will be appreciated,the removable storage unit includes a computer usable storage mediumhaving stored therein computer software and/or data.

Computer system may also include a communications interface.Communications interface allows software and data to be transferredbetween computer system and external devices. In various embodiments,communication between systems may be accomplished over a variety ofnetworks using any suitable communication protocols. These networksand/or communications protocols may include, for example, transport overEthernet, InfiniBand, or wireless, and could follow a variety ofprotocols including TCP, RDMA, or FICON. Remote WAN communication couldalso be done through VPN connections. Remote file systems could bedisplayed as FTP, NFS, or Samba shares. For example, a FICONcommunication protocol may allow the mainframe to write via FICONbecause the FICON may be viewed like it is a mainframe to the mainframecommunication. Moreover, any protocol that the mainframe can support attape speed, mainframe speed, real time speed and/or streaming speedwould be an appropriate protocol to operate the system with the dataprocessing appliance.

The terms “computer program medium” and “computer usable medium” and“computer readable medium” are used to generally refer to media such asremovable storage drive and a hard disk installed in hard disk drive.These computer program products provide software to computer system.

Computer programs (also referred to as computer control logic) arestored in main memory and/or secondary memory. Computer programs mayalso be received via communications interface. Such computer programs,when executed, enable the computer system to perform the features asdiscussed herein. In particular, the computer programs, when executed,enable the processor to perform the features of various embodiments.Accordingly, such computer programs represent controllers of thecomputer system.

In various embodiments, software may be stored in a computer programproduct and loaded into computer system using removable storage drive,hard disk drive or communications interface. The control logic(software), when executed by the processor, causes the processor toperform the functions of various embodiments as described herein. Invarious embodiments, hardware components such as application specificintegrated circuits (ASICs). Implementation of the hardware statemachine so as to perform the functions described herein will be apparentto persons skilled in the relevant art(s).

As used herein an “identifier” may be any suitable identifier thatuniquely identifies an item. For example, the identifier may be aglobally unique identifier (“GUID”). The GUID may be an identifiercreated and/or implemented under the universally unique identifierstandard. Moreover, the GUID may be stored as 128-bit value that can bedisplayed as 32 hexadecimal digits. The identifier may also include amajor number, and a minor number. The major number and minor number mayeach be 16 bit integers.

As used herein, the term “network” includes any cloud, cloud computingsystem or electronic communications system or method which incorporateshardware and/or software components. Specific information related to theprotocols, standards, and application software utilized in connectionwith the Internet is generally known to those skilled in the art and, assuch, need not be detailed herein.

“Cloud” or “Cloud computing” includes a model for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, servers, storage, applications, and services)that can be rapidly provisioned and released with minimal managementeffort or service provider interaction. Cloud computing may includelocation-independent computing, whereby shared servers provideresources, software, and data to computers and other devices on demand.

As used herein, “transmit” may include sending electronic data from onesystem component to another over a network connection. Additionally, asused herein, “data” may include encompassing information such ascommands, queries, files, data for storage, and the like in digital orany other form.

More particularly, a “key field” partitions the database according tothe high-level class of objects defined by the key field. For example,certain types of data may be designated as a key field in a plurality ofrelated data tables and the data tables may then be linked on the basisof the type of data in the key field. The data corresponding to the keyfield in each of the linked data tables is preferably the same or of thesame type. However, data tables having similar, though not identical,data in the key fields may also be linked by using AGREP, for example.In accordance with one embodiment, any suitable data storage techniquemay be utilized to store data without a standard format. Data sets maybe stored using any suitable technique, including, for example, storingindividual files using an ISO/IEC 7816-4 file structure; implementing adomain whereby a dedicated file is selected that exposes one or moreelementary files containing one or more data sets; using data setsstored in individual files using a hierarchical filing system; data setsstored as records in a single file (including compression, SQLaccessible, hashed via one or more keys, numeric, alphabetical by firsttuple, etc.); Binary Large Object (BLOB); stored as ungrouped dataelements encoded using ISO/IEC 7816-6 data elements; stored as ungroupeddata elements encoded using ISO/IEC Abstract Syntax Notation (ASN.1) asin ISO/IEC 8824 and 8825; and/or other proprietary techniques that mayinclude fractal compression methods, image compression methods, etc.

The data set annotation may also be used for other types of statusinformation as well as various other purposes. For example, the data setannotation may include security information establishing access levels.The access levels may, for example, be configured to permit only certainindividuals, levels of employees, companies, or other entities to accessdata sets, or to permit access to specific data sets based on thetransaction, merchant, issuer, user or the like. Furthermore, thesecurity information may restrict/permit only certain actions such asaccessing, modifying, and/or deleting data sets. In one example, thedata set annotation indicates that only the data set owner or the userare permitted to delete a data set, various identified users may bepermitted to access the data set for reading, and others are altogetherexcluded from accessing the data set. However, other access restrictionparameters may also be used allowing various entities to access a dataset with various permission levels as appropriate.

The data, including the header or trailer may be received by astand-alone interaction device configured to add, delete, modify, oraugment the data in accordance with the header or trailer. As such, inone embodiment, the header or trailer is not stored on the transactiondevice along with the associated issuer-owned data but instead theappropriate action may be taken by providing to the transactioninstrument user at the stand alone device, the appropriate option forthe action to be taken. The system may contemplate a data storagearrangement wherein the header or trailer, or header or trailer history,of the data is stored on the transaction instrument in relation to theappropriate data.

One skilled in the art will also appreciate that, for security reasons,any databases, systems, devices, servers or other components of thesystem may consist of any combination thereof at a single location or atmultiple locations, wherein each database or system includes any ofvarious suitable security features, such as firewalls, access codes,encryption, decryption, compression, decompression, and/or the like.

Encryption may be performed by way of any of the techniques nowavailable in the art or which may become available—e.g., Twofish, RSA,El Gamal, Schorr signature, DSA, PGP, PKI, GPG (GnuPG), and symmetricand asymmetric cryptosystems.

The system and method may be described herein in terms of functionalblock components, screen shots, optional selections and variousprocessing steps. It should be appreciated that such functional blocksmay be realized by any number of hardware and/or software componentsconfigured to perform the specified functions. For example, the systemmay employ various integrated circuit components, e.g., memory elements,processing elements, logic elements, look-up tables, and the like, whichmay carry out a variety of functions under the control of one or moremicroprocessors or other control devices. Similarly, the softwareelements of the system may be implemented with any programming orscripting language and with the various algorithms being implementedwith any combination of data structures, objects, processes, routines orother programming elements. Further, it should be noted that the systemmay employ any number of conventional techniques for data transmission,signaling, data processing, network control, and the like.

The system and method is described herein with reference to screenshots, block diagrams and flowchart illustrations of methods, apparatus(e.g., systems), and computer program products according to variousembodiments. It will be understood that each functional block of theblock diagrams and the flowchart illustrations, and combinations offunctional blocks in the block diagrams and flowchart illustrations,respectively, can be implemented by computer program instructions.

These computer program instructions may be loaded onto a special purposecomputer, or other programmable data processing apparatus to produce amachine, such that the instructions that execute on the computer orother programmable data processing apparatus create means forimplementing the functions specified in the flowchart block or blocks.These computer program instructions may also be stored in acomputer-readable memory that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory produce an article of manufacture including instruction meanswhich implement the function specified in the flowchart block or blocks.The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer-implemented process such that theinstructions which execute on the computer or other programmableapparatus provide steps for implementing the functions specified in theflowchart block or blocks.

Accordingly, functional blocks of the block diagrams and flowchartillustrations support combinations of means for performing the specifiedfunctions, combinations of steps for performing the specified functions,and program instruction means for performing the specified functions. Itwill also be understood that each functional block of the block diagramsand flowchart illustrations, and combinations of functional blocks inthe block diagrams and flowchart illustrations, can be implemented byeither special purpose hardware-based computer systems which perform thespecified functions or steps, or suitable combinations of specialpurpose hardware and computer instructions.

The term “non-transitory” is to be understood to remove only propagatingtransitory signals per se from the claim scope and does not relinquishrights to all standard computer-readable media that are not onlypropagating transitory signals per se. Stated another way, the meaningof the term “non-transitory computer-readable medium” and“non-transitory computer-readable storage medium” should be construed toexclude only those types of transitory computer-readable media whichwere found in In Re Nuijten to fall outside the scope of patentablesubject matter under 35 U.S.C. § 101.

Benefits, other advantages, and solutions to problems have beendescribed herein with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any elements that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as critical, required, or essentialfeatures or elements of the disclosure. The scope of the disclosure isaccordingly to be limited by nothing other than the appended claims, inwhich reference to an element in the singular is not intended to mean“one and only one” unless explicitly so stated, but rather “one ormore.” Moreover, where a phrase similar to ‘at least one of A, B, and C’or ‘at least one of A, B, or C’ is used in the claims or specification,it is intended that the phrase be interpreted to mean that A alone maybe present in an embodiment, B alone may be present in an embodiment, Calone may be present in an embodiment, or that any combination of theelements A, B and C may be present in a single embodiment; for example,A and B, A and C, B and C, or A and B and C. Although the disclosureincludes a method, it is contemplated that it may be embodied ascomputer program instructions on a tangible computer-readable carrier,such as a magnetic or optical memory or a magnetic or optical disk. Allstructural, chemical, and functional equivalents to the elements of theabove-described various embodiments that are known to those of ordinaryskill in the art are expressly incorporated herein by reference and areintended to be encompassed by the present claims.

Moreover, it is not necessary for a device or method to address each andevery problem sought to be solved by the present disclosure, for it tobe encompassed by the present claims. Furthermore, no element,component, or method step in the present disclosure is intended to bededicated to the public regardless of whether the element, component, ormethod step is explicitly recited in the claims. No claim element hereinis to be construed under the provisions of 35 U.S.C. 112 (f) unless theelement is expressly recited using the phrase “means for.” As usedherein, the terms “comprises”, “comprising”, or any other variationthereof, are intended to cover a non-exclusive inclusion, such that aprocess, method, article, or apparatus that comprises a list of elementsdoes not include only those elements but may include other elements notexpressly listed or inherent to such process, method, article, orapparatus.

What is claimed is:
 1. A method, comprising: receiving, by a dataprocessing appliance, a workflow from at least one of a mainframecomputing resource or a distributed computing resource; receiving, bythe data processing appliance, data from at least one of the mainframecomputing resource or the distributed computing resource; preparing, bythe data processing appliance and in real-time as the data is received,the data for at least one of the mainframe computing resource or thedistributed computing resource; streaming, by the data processingappliance and contemporaneously with the preparing, the data for atleast one of the mainframe computing resource or the distributedcomputing resource; receiving, by the a data processing appliance, thefirst data that has been processed in a second format by the adistributed computing resource; converting, by the data processingappliance, the processed first data from the second format to a firstformat; writing, by the data processing appliance and contemporaneouslywith converting the first data, the first data in the first format to afirst destination corresponding to at least one of a data library or amainframe computing resource; receiving, by the data processingappliance, the second data that has been processed in the first formatby the mainframe computing resource; converting, by the data processingappliance, the processed second data from the first format to the secondformat; and writing, by the data processing appliance andcontemporaneously with the converting the second data, the processedsecond data in the second format to a second destination correspondingto at least one of a the data library or the distributed computingresource, wherein: the mainframe computing resource is configured tocommunicate communicates securely with the data library through a firstinterface and the data processing appliance is configured to communicatecommunicates with the data library through a second interface, and theprocessed second data in the second format is stored in a first portionof the data library that is segregated from a second portion of the datalibrary that stores received second data that had been processed in thefirst format by the mainframe computing resource.
 2. The method of claim1, wherein the data processing appliance comprises at least one of aplurality of hardware accelerators, or a plurality of computerprocessing units.
 3. The method of claim 2, wherein the plurality ofhardware accelerators comprises at least one of a plurality of logicgates or a plurality of graphical processing units, and the dataprocessing appliance is configured to parse parses the first data andthe second data for parallel processing of portions of the first dataand the second data by the plurality of logic gates, the plurality ofgraphical processing units, and the plurality of computer processingunits.
 4. The method of claim 2, wherein the plurality of hardwareaccelerators comprises at least one of a plurality of logic gates or aplurality of graphical processing units.
 5. The method of claim 1,wherein the preparing includes preparing the first data in is convertedto the first format in response to the first data being transmittedwritten to the mainframe computing resource or preparing the second datain is converted to the second format in response to the second databeing transmitted written to the distributed computing resource.
 6. Themethod of claim 5, wherein the mainframe computing data processingappliance is configured to stream streams the first data in at least oneof the first format to the mainframe computing appliance or resource andthe second data in the second format to the distributed computingresource.
 7. The method of claim 1, wherein the first interface isisolated form from the second interface.
 8. A system comprising: a dataprocessing appliance having a processor, a tangible, non-transitorymemory configured to communicate with the processor, the tangible,non-transitory memory having instructions stored thereon that, inresponse to execution by the processor, cause the processor to becapable of performing operations comprising: receiving, by the dataprocessing appliance, a workflow from at least one of a mainframecomputing resource or a distributed computing resource; receiving, bythe data processing appliance, data from at least one of the mainframecomputing resource or the distributed computing resource; preparing, bythe data processing appliance and in real-time as the data is received,the data for at least one of the mainframe computing resource or thedistributed computing resource; streaming, by the data processingappliance and contemporaneously with the preparing, the data for atleast one of the mainframe computing resource or the distributedcomputing resource; receiving, by the a data processing appliance, thefirst data that has been processed in a second format by the adistributed computing resource; converting, by the data processingappliance, the processed first data from the second format to a firstformat; writing, by the data processing appliance and contemporaneouslywith the converting the first data, the processed first data in thefirst format to a first destination corresponding to at least one of adata library or the a mainframe computing resource:; receiving, by thedata processing appliance, the second data that has been processed inthe first format by the mainframe computing resource; converting, by thedata processing appliance, the processed second data from the firstformat to the second format: and writing, by the data processingappliance and contemporaneously with the converting the second data, theprocessed second data in the second format to a second destinationcorresponding to at least one of a the data library or the distributedcomputing resource, wherein: the mainframe computing resource isconfigured to communicate communicates securely with the data librarythrough a first interface and the data processing appliance isconfigured to communicate communicates with the data library through asecond interface, and the processed second data in the second format isstored in a first portion of the data library that is segregated from asecond portion of the data library that stores received second data thathad been processed in the first format by the mainframe computingresource.
 9. The system of claim 8, wherein the data processingappliance comprises at least one of a plurality of hardwareaccelerators, or a plurality of computer processing units.
 10. Thesystem of claim 9, wherein the plurality of hardware acceleratorscomprises at least one of a plurality of logic gates or a plurality ofgraphical processing units, and the data processing appliance isconfigured to parse parses the first data and the second data forparallel processing of portions of the first data and the second data bythe plurality of logic gates, the plurality of graphical processingunits, and the plurality of computer processing units.
 11. The system ofclaim 9, wherein the plurality of hardware accelerators comprises atleast one of a plurality of logic gates or a plurality of graphicalprocessing units.
 12. The system of claim 8, wherein the preparingincludes preparing the first data in is converted to a first format inresponse to the first data being transmitted written to the mainframecomputing resource or preparing the second data in is converted to asecond format in response to the second data being transmitted writtento the distributed computing resource.
 13. The system of claim 12,wherein the mainframe computing data processing appliance is configuredto stream streams the first data in at least one of the first format tothe mainframe computing appliance or resource and the second data in thesecond format to the distributed computing resource.